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Background 


•  Emergence  of  new  buzzwords  in  software  development 

-  Competitive  pressures  of  the  1990s  forced  software  companies  to  reexamine 
their  development  processes  and  adopt  radical  approaches.  As  a  result ,  the 
industry  has  been  flooded  with  buzzwords  like  “internet  time/’  “ extreme ,  ”  and 
“agile,” just  to  mention  a  few 

•  Management  buzzwords  have  been  flooding  over  the  past  30  years... 

-  There  has  been  a  “bandwagon  effect”  of  popular  management  movements 
such  as  total  quality  management  (TQM),  management  by  objectives, 
reinventing  government,  reengineering,  the  balanced  scorecard,  lean,  and 
Six  Sigma®.  However, 

•  companies  that  claimed  excellence  on  the  basis  of  these  practices  later 
turned  out  to  be  mediocre  or  outright  failures  [Paparone  2009] 

-  Consequently,  a  relatively  recent,  interesting  recommendation  to  the 
Pentagon  brass:  “Stay  away  from  management  bestsellers...” [Erwin  2009] 


*  Six  Sigma  has  been  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Motorola 

0 AEROSPACE 
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Motivation 


•  History  notwithstanding... 

-  Agility  seems  to  be  a  simple  concept 

-  It  is  commonly  perceived  as  a  virtue 

-  Agile  methods  are  making  inroads  into  software  development 

•  Despite  of  Ms.  Erwin’s  advice,  Pentagon  brass  does  not  seem  to  be 
able  to  stay  away  from  management  bestsellers  after  all  © 

•  Consequently,  the  idea  of  bringing  agile  concepts  into  defense 
acquisition  requires  a  closer  look 
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Objectives 


•  Readers  will  be  able  to 

-  Name  popular  agile  software  development  methods 

-  Describe  representative  agile  software  development  practices 

-  Compare  agile  and  traditional  development  methods 

-  Assess  the  appropriateness  of  an  organization’s  software  development 
practices 

-  Appreciate  the  spirit  and  usefulness  of  mission  assurance  in  carrying  out  the 
evaluations  of  the  defense  contractors’  software  development  practices 

-  Differentiate  between  agility  in  acquisition  and  agility  in  development 
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What  is  Agility? 

The  narrow,  dictionary  definition  [Collins  2012]: 

-  Quick  in  movement;  nimble 

Agility  implies  both  the  capacity  and  capability  to  act  immediately 

-  Agility  is  perceived  a  virtue 

-  In  business,  agility  is  considered  an  important  organizational  capability 
Unfortunately,  in  most  contexts  it  is  ill-defined  or  inconsistent 

-  Agility  does  not  simply  equate  with  speed,  as  the  following  examples  show 

•  Agility  may  conflict  with  speed 

-  The  Titanic’s  ability  to  turn  sharply  is  far  more  likely  to  avert  disaster 
than  increasing  its  top  speed  charging  straight  ahead 

•  Agility  requires  speed  but  also  requires  balance 

-  e.g.,  in  martial  arts 

-  “Lean”  does  not  always  equate  with  “agile” 

•  e.g.,  applying  lean  concepts  might  increase  the  rigidity  of  a  process 

-  This  rigidity  results  from  constraining  the  process  in  order  to  optimize 
the  case  “right  now” 

Agility  is  like  the  Elixir  of  Life  or  the  Fountain  of  Youth  -  Mysterious  and  Elusive 

-Anonymous 
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Agility  in  Defense 


•  The  warfighter  perspective 

-  A  confusion  exists  about  the  need  for  systems  enabling  warfighter 
agility  vs.  the  need  for  agile  acquisition  of  weapon  systems 

•  No  argument  about  the  value  of  warfighter  agility.  However, 

-  Warfighter  agility  can  be  primarily  supported  via  weapons 
design  and  flexible  architecture 

-  Faster  access  to  new  weapons  is  not  always  the  right  solution 

-  The  tradeoff  between  faster  access  and  features  is  promoted, 
but  the  underlying,  hidden  quality  concessions  are  always 
controversial  and  the  associated  decisions  are  very  difficult 

•  The  acquisition  perspective 

-  Essential  concerns  exist  that  need  to  be  clarified  and  answered 

•  To  what  extent  would  agile  software  development  contribute  to 
the  achievement  of  agile  acquisition  of  weapon  systems? 

•  How  is  fast  procurement  different  from  agile  acquisition? 

•  Under  what  circumstances  is  agile  software  development 
acceptable  or  even  desirable  for  weapon  systems  acquisition? 


For  operational  responsiveness  we  need  “agile  products”  and  not  “agile  processes”  I 

0 AEROSPACE 


(Our)  Definition  of  Agile  Software  Development 


•  Agile  software  development  methods  employ  practices  that  are 
consistent  with  the  Agile  Manifesto’s  value  statements  and  principles 

-  There  are  numerous,  “brand-name”  methods  that  are  considered  agile* 

-  However,  “new”  approaches  are  published  almost  every  day  that  are  mostly 
mix-and-match  medleys  of  existing  practices 

•  History  of  the  Agile  Manifesto** 

-  Created  on  February  11-13,  2001  at  the  first  meeting  of  agile  proponents, 
the  17  founding  members  of  the  Agile  Alliance 

•  Agile  values: 

-  “We  are  uncovering  better  ways  of  developing  software  by  doing  it  and 
helping  others  doing  it.  Through  this  work  we  have  to  come  to  value: 

•  Individuals  and  interactions  over  processes  and  tools 

•  Working  software  over  comprehensive  documentation 

*  Customer  collaboration  over  contract  negotiation 

*  Responding  to  change  over  following  a  plan." 
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*  See  the  backup  charts;  **  For  the  complete  text  see  [Agile  2001] 
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The  Agile  Manifesto  Principles 


•  The  following  principles  are  used  to  select  development  practices 

(1)  Early  and  continuous  delivery  to  satisfy  customers 

(2)  Welcoming  changing  requirements 

(3)  Delivering  working  software  frequently 

(4)  Close  collaboration  with  business  people 

(5)  Motivation  of  developers  through  trust 

(6)  Using  face-to-face  conversations  to  convey  information 

(7)  Working  software  is  the  primary  measure  of  progress 

(8)  Sponsors,  developers,  and  users  maintain  a  constant  pace 

(9)  Continuous  attention  to  good  design 

(10)  Simplicity,  maximizing  the  amount  of  work  not  done 

(11)  The  best  work  is  always  expected  from  self-organizing  teams 

(12)  Team  reflection  and  behavior  adjustment  at  regular  intervals 
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In  Contrast,  Principles  of  Modern  Software  Management 


•  “Modern”  software  management  predates  the  Agile  Manifesto 

-  However,  its  principles  are  drastically  different  from  the  “ traditional waterfall 
development 

-  Modern  management  is  indeed  plan-based,  process  and  tools-focused* 

•  Modern  software  management  principles 

(1)  Architecture-first  approach 

(2)  Iterative  life-cycle  process 

(3)  Component-based  development 

(4)  Establish  a  change  management  environment 

(5)  Enhance  change  freedom  through  tools  that  support  round-trip  engineering 

(6)  Rigorous,  model-based  notation 

(7)  Objective  quality  control 

(8)  Demonstration-based  approach  to  assess  intermediate  artifacts 

(9)  Intermediate  releases  with  evolving  levels  of  detail 

^ddit!onaMy^T3lUiee^^pu^oftwar^evelopmennMh^cquis!tioi^ontextl 

•  Source:  [Royce  1 998] 
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Assuring  Space  Mission  Success 


Still  Flying  High  -  Context  and  Building  Blocks 


Defense  Acquisition  (The  Big  “A”  Acquisition  Process...) 
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Agile  Software  Development  of  New  Weapon  Systems 


JCIDS 


Allied 

Capabilities 


PPBE 


oso, 

White  House 


Congress 

Branch) 


Legend: 

DOD  Department  of  Defense 

JCIDS  Joint  Capabilities  Integration  &  Development  System 
JROC  Joint  Requirements  Oversight  Council 
OSD  Office  of  the  Secretary  of  Defense 
PPBE  Planning,  Programming,  Budgeting  &  Execution 

R  Performance  &  "Time  to  Need"  Requirements 

$  Allocated  Funding 


Agile  software  development  affects  only  the  smaller  context  of  DOD  5000.02 
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Key  Stakeholders  in  the  Big  “A”  Acquisition  Process 


JCIDS 


Actual 

Customer 


ODD  5000.02 


Legend; 

DOD  Department  of  Defense 

JCIDS  Joint  Capabilities  Integration  &  Development  System 
JROC  Joint  Requirements  Oversight  Council 
OSD  Office  of  the  Secretary  of  Defense 
PPBE  Planning,  Programming,  Budgeting  &  Execution 

R  Performance  &  "Time  to  Need"  Requirements 

$  Allocated  Funding 


Oversight  - 
Organizations 

( Ac  qu  liUtei  pi  WoHif 

11 

i 

Ion  tractor  | 

Surrogate  User, 
Surrogate  Customer 


Management 


Note  how  removed  development  is  from  the  actual  user  and  customer 
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•  Because  of  different  motivation  and  behavior,  there  is  a 
tension  between 


Acquisition  is  a  Contact  Sport... 


-  Stakeholders  of  the  acquisition  process  (e.g.,  Congress,  DOD,  etc.) 

-  Stakeholders  of  the  oversight  organizations  (e.g.,  acquisition  program  offices 
(APOs*),  and  the  development  organizations  (contractors) 

-  Stakeholders  of  the  development  organizations  themselves 

•  Management  vs.  developers 

•  Hardware  developers  vs.  software  developers 

•  Some  hard  facts  to  face 

-  Typically  the  conflicts  are  not  between  equals 

-  Different  stakeholders  have  different  political  weight  and  capabilities,  hence 
in  most  cases  “ win-win ”  solutions  are  either  not  feasible  or  not  pursued 

•  New  valuation  considerations  for  agile  software  development  practices 

-  Potential  impact  on  existing  tensions  in  the  overall  acquisition  system 

-  Loyalty  factor,  i.e.,  whose  interest  should  be  acknowledged  as  the  most 
important  in  a  particular  context 

Th^undamenta^ourc^Hensioin^whicI^takehoide^il'^eaMheTisI^ 

*APO  is  a  generic  term;  program  offices  are  called  differently  in  different  services 


0 AEROSPACE 
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The  Risk  Pendulum  -  Who  is  Going  to  Bear  the  Risks? 


Basic  Funding  Patterns* 

Cost-based 

Time-based 

Fixed  Price 

Promise 

Best  effort 

Best  effort 

Shall  deliver 

Cash  flow  As  incurred  As  incurred  On  delivery  of  item 


Customer  control 

Maximal 

Maximal 

Minimal 

Risk  to  contractor  or  developer 

Low 

Low 

High 

Risk  to  customer  or  management 

High 

High 

Low 

*  Note  that  these  patterns  have  their  formal, 
contracting  equivalents  and  variations  in 
the  Federal  Acquisition  Regulation  (FAR) 


Customer  or 
management 


Risk  Pendulum 


Contractor  or 
developer 


< 


18 


The  interesting  paradox  is  that  despite  higher  customer  control  -  which  is  perceived  to 
drive  down  risk  -  cost-based  and  time-based  patterns  are  still  risky. . .  (2)  AEROSPACE 


Clarifying  Loyalties 


•  Actual  users  and  the  customer  are  far  removed  from  actual 
development 

•  The  primary  stakeholders  we  need  to  help  are  the  people  in  APOs 

-  They  play  the  complex  role  of  both  surrogate  user  and  surrogate  customer 
by 

•  Providing  technical  input  as  surrogate  user 

•  Providing  contract  management  as  surrogate  customer 

•  The  main  objective  of  these  primary  stakeholders  is  mission  success 

-  Of  course,  this  is  not  different  from  the  actual  users’  and  actual  customer’s 
objective 

•  However,  only  they  have  the  direct,  tactical  means  via  mission 
assurance 
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Mission  Assurance  Definitions* 


•  Mission  Success 

-  The  achievement  by  an  acquired  system  (or  system  of  systems)  to  singularly 
or  in  combination  meet  not  only  specified  performance  requirements  but  also 
expectations  of  users  and  operators  in  terms  of  safety,  operability,  suitability, 
and  supportability 

-  Mission  success  is  evaluated  after  operational  turnover,  according  to 
program-specific  timelines  and  criteria 

•  Mission  Assurance 

-  The  disciplined  application  of  general  systems  engineering,  quality,  and 
management  principles  towards  the  goal  of  achieving  mission  success,  and 
towards  this  goal,  this  disciplined  application  provides  confidence  in  its 
achievement 
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*  Source:  [Guarro  2007] 
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Mission  Assurance  is  Development  Process-neutral 


•  Software  mission  assurance  does  not  assume  any  particular  software 
development  methodology,  programming  language,  or  tools 

•  Mission  assurance  is  the  exclusive  responsibility  of  the  APO,  a  defense 
acquisition  oversight  organization 

-  Note  that  Air  Force  APO’s  enjoy  direct  help  from  multiple  entities,  such  as 

•  Federally  Founded  Research  and  Development  Centers  (FFRDCs) 

•  Systems  Engineering  and  Technical  Assistance  (SETA)  contractors 

•  Systems  Engineering  &  Integration  (SE&I)  contractors 

•  The  APO’s  mission  assurance  activities  do  not  assume  the  presence  of 
any  similar,  or  similarly  named  (i.e.,  “Mission  Assurance”)  effort  from  the 
contractor 

-  If  such  effort  exists  then,  from  the  APO’s  perspective,  it  needs  to  be  treated 
as  an  integral  part  of  the  contractor’s  software  development  process 


Software  mission  assurance  tasks  are  inherently  essential  for  the  assurance 

of  any  software  development  endeavor  in  defense  acquisition  | 

(%)  AEROSPACE 
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The  Main  Exposure  to  Mission  Success:  Software  Defects 


•  Definition  of  a  software  defect 

-  Any  software  attribute  or  characteristic  that  represents  a  deviation  from 
specified  attributes  or  characteristics 

-  Software  defects  can  cause  unanticipated  cost  and  schedule  overruns  and  in 
operational  systems  performance  deficiencies 

•  Definition  of  a  software  fault 

-  A  software  fault  is  a  software  defect  that  can  result  in  a  significant  system 
function  failure  during  the  execution  of  the  code 

•  Hardware-induced  vs.  software-induced  failures 

-  Hardware-induced  failures 

•  Software  always  depends  on  hardware;  certain  hardware  defects  might 
manifest  themselves  as  software  defects  (e.g.,  a  single-event  upset 
(SEU)  in  the  onboard  computer’s  memory  or  registers  as  a  result  of 
naturally  occurring  cosmic  rays,  trapped  protons,  and  solar  energetic 
particles) 

-  Software-induced  failures 

•  Majority  of  such  failures  are  rooted  in  software  design  or  specification 
flaws;  essentially  the  system  enters  into  an  unanticipated  and/or  poorly 
understood  operational  regime 
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*  Definitions  courtesy  of  Myron  Hecht  [Guarro  2008] 
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Software-induced  Failure*  Types 


Pssssstll! 


•  Deterministic  vs.  random  failures 

-  Deterministic  (“Bohrbugs”) 

•  Repeatable 

•  Traceable  to  root  cause(s)  under  control  of  developer  or  user 

-  Deterministic  failures  can  be  prevented  through  the  use  of  a 
disciplined  development  process 

-  Random  (“Heisenbugs”) 

•  Not  repeatable;  many  failures  can  be  fixed  by  reset 

•  Caused  by  transient  states  of  the  software  (timing,  buffer  overflows, 
queues,  memory  leaks,  etc.) 

•  Indistinguishable  from  SEUs,  power  fluctuations,  or  hardware  timing 
errors 

•  Recoverable  vs.  non-recoverable  software  failures  (space  example) 

-  Recoverable  software  failures  are  events  that  occur  in  spacecraft  processors 
that  cause  a  loss  or  performance  degradation  of  the  bus  or  payload,  which 

^at^^estore^i^ithe^nboar^oi^noun^oiTectiv^ction^^^^ 

Application  of  a  disciplined  development  process  itself  is  not  a  guarantee 
for  preventing  random  failures  or  mitigating  recoverable  failures 


*  Forsake  of  simplicity  they  will  be  referenced  as  software  failures 
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Preventing  Random  Software  Failures 


•  The  following  approach  is  recommended* * 

-  Collect  software  failure  data  during  integration  testing 

•  Use  relevant  operational  profiles,  not  just  requirements,  to  define  test 
plans 

•  Record  software  operating  time 

•  Record  all  failure  events 

•  Collect  recovery  time  and  data  to  determine  the  probability  of  recovery 

-  Select  an  appropriate  software  reliability  model 

•  This  model  will  be  used  to  extrapolate  behavior  from  test  data 

-  Evaluate  parameters 

•  Software  behavior  must  be  analyzed  and  validated  via  formal,  systematic 
means  that  take  into  account  a  variety  of  nominal  and  off-nominal 
operational  scenarios 

-  Integrate  findings  into  the  appropriate  system  stochastic  or  reliability  model 

Most  likely  the  contractors  use  similar,  complex  models;  verifying  the 
correctness  of  the  contractors’  analyses  is  a  critical  mission  assurance  task 

*  Source:  [Guarro  2008] 
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The  Life  Cycle  Perspective  of  Agile  Software  Development 


0 AEROSPACE 
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Agile  Life  Cycle  Example:  Scrum* 


•  Scrum  is  a  lean  approach  to  software  development 

-  Simple  “inspect  and  adapt”  management  framework,  using  time-boxing 

-  Based  on  the  scrum  metaphor  for  new  product  development  [T akeuchi  1986] 

-  No  declared,  method-specific  development  practices 

-  “ Backlog ”  is  a  metaphor  for  requirements 

*  The  process  was  first  formalized  by  Ken  Schwaber  [Schwaber  95] 


In  Contrast,  an  Iterative-Incremental  Life  Cycle,  IBM/RUP 


IBM/RUP  Phases 


Core  Workflows 

Requirements 

Analysis 

Design 

Implementation 

Test 

Deployment 


The  Rational  Unified  Process  (RUP)  is  a  comprehensive  process  model* 

-  Workflows  are  essentially  life  cycle  processes  with  detailed  descriptions 

-  The  process  encompasses  the  earlier  outlined,  “modern”  principles  [Royce  1998] 

-  It  has  been  renamed  IBM/RUP  after  the  acquisition  of  Rational  Corp.  by  IBM 


0)  AEROSPACE 
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*  Discussion  is  based  on  [Jacobson  1 999] 


After  We  Remove  the  Fluff  (i.e.,  the  Metaphors...) 


Iterative-Incremental  Development  (IID) 

Content  (Requirements)  Driven 


Time-box 

Calendar  (“Clock”)  Driven 


Factors  to  be  compared  IID 


Iteration/Increment  duration 
Iteration  content  in  the  context  of  an  increment 

Difficulty  of  iteration  planning 

Difficulty  of  increment  planning 
Micro-estimation  fidelity 
Macro-estimation  fidelity 
Naturally  fitting  contracting  pattern 


varying 

planned  upfront 
moderate 
difficult 
moderate 
high 

cost-based 


Time-box 


set 

not  planned  upfront 
easy 

difficult 


higher  than  IID 

low 

time-based 


Red  flag  marks  the  customers’  primary  concerns 
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Agile  Software  Development  Values 
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Examining  Agile  Software  Development  Values 


•  Agile  software  development  values  revisited 

-  Individuals  and  interactions  over  processes  and  tools 

-  Working  software  over  comprehensive  documentation 

-  Customer  collaboration  over  contract  negotiation 

-  Responding  to  change  over  following  a  plan 

•  During  the  analysis  the  following,  typical  figures  should  be  considered 

-  Space  vehicle  (embedded,  large,  including  bus  software  and  payload(s)): 

•  -512  thousand  delivered  source  instructions  (KDSI) 

-  Ground  systems: 

•  Space  Shuttle  software  -2,000  KDSI 

•  Satellite  control  systems  software  -4,700  KDSI 

-  The  mentioned  space  vehicle  software  development  of  512  KDSI  would 
require  roughly  a  6,420  person-month  effort,  spreading  over  41  months, 
involving  -157  full-time  equivalent  software  personnel 
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Individuals  and  Interactions  Over  Processes  and  Tools 


•  Let’s  focus  on  processes  first 

-  Agile  proponents  believe  that  one  should  only  declare  and  rely  on  practices 
instead  of  processes  to  increase  the  agility  of  software  development 

•  A  practice  usually  refers  to  an  individual  activity  while  a  process  is  an 
aggregate  structure  of  multiple  activities 

-  Relying  only  on  practices  certainly  ensures  a  greater  level  of  flexibility, 
however: 

•  This  flexibility  comes  with  unavoidable  ambiguities  and  may  create 
tension  among  the  stakeholders 

-  Consider  the  example’s  157  developers  working  shoulder-to-shoulder 

-  Consider  the  problems  of  concurrent  hardware-software  development 

-  In  pursuing  mission  success  we  found  that  even  the  use  of  so-called  mature 
processes,  such  as  defined  by  the  CMMI®,  proved  to  be  inadequate 

The  government  must  make  a  robust  software  standard  contractually  compliant 

[Eslinger  2006] 

®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University 
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Lean 

•  The  term  “lean  production”  was  coined  in  the  80’s  [Krafcik  1988] 

-  The  underlying  ideas  represent  the  so-called  lean  thinking  about  processes 

•  Current  (mis)use  of  the  term 

-  Lean  is  a  popular  buzz-word  for  general  cost-cutting  efforts 

-  Lean  may  be  used  in  conjunction  with  Six  Sigma®,  another,  also 
manufacturing-rooted,  process  improvement  method  (“Lean  Six  Sigma") 

•  Unfortunately,  this  term  is  misleading:  “lean”  does  not  mean  applying  lean 
thinking  to  Six  Sigma  but  using  Six  Sigma  tools  to  carry  out  lean  practices 

•  Key  principles  of  lean  systems  thinking  [Rule  201 1] 

-  Understand  value  from  the  stakeholders’  perspective 

-  Identify  all  steps  in  the  value  stream 

-  Enable  value  to  flow  smoothly 

-  Respond  to  the  pull  of  stakefiglder  demand 

-  Continuously  seek  perfection 

•  Mission  assurance  exposure  ^ 

-  Difficult  to  sort  out  what  is  really  important  due  to  stakeholder  conflicts  ” 

-  Lean  Six  Sigma  rule  of  thumb  is  that  usually  only  5%  of  total  process  cycle 
time  adds  value  to  outputs;  mission  assurance  is  valued  low  by  developers 


®  Six  Sigma  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Motorola 
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Major  Areas  in  a  Typical  Software  Development  Standard* 


System  and  Software  (SW)  Architecture 

Human  Systems  Integration 

Interoperability  and  Standardization 

Reliability,  Safety,  Information 
Assurance 

Project  Planning  and  Oversight 
SW  Development  Environment 
System  Requirements  Analysis 
SW  Requirements  Analysis 
SW  Design 

SW  Implementation  and  Unit  Testing 
Unit  Integration  and  Testing 
SW  Qualification  Testing 


The  “lean”  question:  Which  ones  do 


Transition  to  Operations  and 
Maintenance 

SW  Configuration  Management 

SW  Peer  Review/Product  Evaluation 

SW  Quality  Assurance 

Corrective  Action 

Joint  Technical  and  Management 
Reviews 

Risk  Management 

SW  Management  Indicators  (Metrics) 

Security  and  Privacy 
Subcontractor  Management 
Interface  with  SW  IV&V  Agents 


add  value?  Which  ones  to  get  rid  off? 


*  Source:  [Adams  2005] 
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What  Does  My  Dentist  Know  About  Mission  Assurance? 


Sign  in  my  dentist’s  office: 

“Brush  only  those  teeth  you  wish  to  keep...” 
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Individuals  and  Interactions  Over  Processes  and  Tools-2 


•  Tools 

-  The  typical  3-4  year  long  development  and  a  minimum  5-10  year  long 
operation  and  sustainment  for  a  space  vehicle  require  strong  tools  support 

•  Development  must  be  based  on  an  architecture-first  approach 

-  Architecture  modeling  artifacts  need  to  be  documented  with  rigorous 
notation  and  handled  with  appropriate  (preferably  visual)  modeling 
tools 

-  The  dynamics  of  concurrent  workflows  by  different  teams  working  on 
shared  artifacts  necessitates  a  rigorously  controlled  change 
management  environment 

•  Tools  are  also  necessary  to  keep  all  the  engineering  information  in 
different  formats  synchronized  and  to  support  bidirectional  traceability 

-  System  requirements,  software  specifications,  design  models,  source 
code,  executable  code,  scripts,  test  cases,  test  data,  etc. 

•  True  change  freedom  cannot  be  realistically  achieved  without  the 
support  of  an  appropriate,  integrated  environment  [Royce  1998] 

^vernr^^tabl^abo^orcMaciUoTowIedg^haringl^io^ufficien^TI 
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Work  Force  Volatility 

•  The  work  force  in  the  information  sector  is  very  volatile* *  even  during 
recessions  when  the  overall  net  employment  change  is  lower  than  average 


Periods  of 

Information  Sector 

Federal  Sector 

Recession** 

Hires 

Separations 

Hires 

Separations 

2001-2002 

36.5% 

43.3% 

19.75% 

19.4% 

2008-2010 

23.7% 

27.8% 

22.13% 

21.0%  1 

•  How  to  interpret  the  data 

-  Unfortunately,  the  Bureau  of  Labor  and  Statistics  (BLS)  is  not  collecting  the 
exact  data  we  would  be  interested  in,  i.e.,  programming-related  turnover  in  the 
defense  industry 

-  However,  one  can  see  that  the  turnover  rate  is  quite  high  even  in  the  federal 
sector,  which  is  considered  less  volatile  than  the  private  sectors 

-  Additionally,  the  BLS  database  does  not  track  internal  company  turnover 


Insisting  on  tacit  knowledge  sharing  is  inappropriate 
in  case  of  such  a  volatile  work  force 


*  Source:  Bureau  of  Labor  and  Statistics  database;  **  [Bruy ere  2011] 


Working  Software  Over  Comprehensive  Documentation 


•  Agile  proponents  essentially  do  not  dispute  that  documentation  plays  an 
important  role  in  software  development  [Ambler  2011] 

-  Author  makes  a  point  from  an  agile  perspective  that  customers  must 
understand  the  total  cost  of  ownership  (TCO)  for  a  document,  and  they  must 
explicitly  decide  to  invest  in  that  document 

-  This  a  good  advice  under  any  circumstances,  of  course 

•  However,  this  value  statement  is  about  interim  progress  assessment 

-  The  idea  is  not  new;  modern  processes  are  already  using  the  demonstration- 
based  approach  to  assess  intermediate  artifacts  [Royce  1998] 

•  The  concern  regarding  the  agile  approach  is  the  impact  on  the  customer 

-  Principle  # 8  of  the  Agile  Manifesto  represents  a  strong  imposition  on  the 
customer:  “Sponsors,  developers,  and  users  maintain  a  constant  pace” 
Unfortunately,  maintaining  such  a  pace  is  not  feasible  on  large  projects 

-  Issues: 

•  Embedding  users/customers  with  the  necessary  expertise  into  every  team 

•  Users/customers  need  to  approve  technical  decisions  in  the  short  cycles 

•  Coordination  of  an  extensive  network  of  user/customer  representatives 

T^hort^hi^gn^alu^oe^io^cal^i^r^narg^rojec^ 

0 AEROSPACE 


38 


Customer  Collaboration  Over  Contract  Negotiation 


•  As  it  was  shown,  actual  users  and  customers  are  far  removed  from  the 
development  organization 

-  JROC,  DOD,  and  Congress  are  high-inertia  organizations  with  complex , 
bureaucratic  processes  for  interaction 

-  These  are  stakeholders  with  different  political  weights;  building  true 
collaborative  relationships  is  difficult  if  not  impossible 

•  With  the  current,  rigid  “upstream”  relationship  the  flexibility  of  the 
surrogate  customer  is  very  limited 

-  Agile  development  will  not  improve  the  agility  of  the  acquisition  process;  in 
fact,  insisting  on  developer  agility  may  exacerbate  the  existing  tensions 

•  It  is  an  unfortunate  fact  of  life  that  when  things  do  not  go  well, 
collaborative  resolution  becomes  less  and  less  feasible 

-  The  stakeholders  have  their  own,  different  risk  perspectives  and  motivations 
and  their  differences  cannot  be  easily  reconciled  via  voluntary  actions 

•  You  would  not  remodel  your  kitchen  without  a  detailed  contract,  so  why 
would  you  deemphasize  the  importance  of  contracts  for  billion-dollar 
weapon  system  acquisitions? 

-  We//,  actually  we  did  it  in  the  1990s;  it  was  called  “ Acquisition  Reform” 

0 AEROSPACE 


39 


Responding  to  Change  Over  Following  a  Plan 


The  essential  motivation  is  the  recognition  that  solution  details  to  complex 
problems  cannot  be  successfully  determined  upfront 

-  This  is  not  a  new  idea;  that’s  why  modern,  but  pre-agile  software  development 
methods  are  adaptive  and  use  iterative/incremental  processes.  How 
requirements  risks  are  handled  in  modern  methods: 

•  On  micro-level:  The  emphasis  during  the  planning  of  iterations  is  on 
facilitating  a  successively  refined  understanding  of  requirements 

•  On  macro-level:  New  or  changing  requirements  are  expected  to  be  handled 
via  evolutionary  acquisition  and  development  strategies 

Agile  principle  #2  (“Welcoming  changing  requirements’’)  is  directly 
flowing  from  the  discussed  agile  value  statement 

-  Unfortunately,  this  is  a  disingenuous  statement,  to  say  the  least 

•  In  reality,  everybody  likes  to  work  on  stable  grounds  with  clear,  unchanging 
expectations;  Don’t  you? 

However,  if  anyone  still  has  doubts,  listen  to  Yogi  Berra: 


“If  you  don't  know  where  you  are  going,  you  will 
wind  up  somewhere  else” 
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Beyond  Unavoidable  Requirements  Volatility 


•  Even  though  Yogi  Berra  was  right,  a  certain  level  of  requirements  volatility 
is  unavoidable 

-  Consequently,  whatever  process  is  used,  some  level  of  flexibility  is  needed  to 
deal  with  such  volatility 

•  However,  lack  of  control  may  still  lead  to  the  erosion  of  process  discipline 

-  " Just  because  you  have  a  detailed  requirements  specification  that  has  been 
reviewed  and  signed  off,  that  doesn't  mean  that  the  development  team  will 
read  it,  or  if  they  do,  that  they  will  understand  it,  or  if  they  do,  that  they  will 
choose  to  work  to  the  specification."  ~~~  Scott  W.  Ambler  [Ambler  2007] 


Only  diligent  mission  assurance  can 


prevent  this  from  happening 
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extreme  Programming 
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extreme  Programming  (XP)* * 


•  What  is  extreme  Programming? 

-  XP  is  a  lightweight,  low-ceremony  software  development  methodology 

•  Based  on  Kent  Beck’s  early  experiences  at  Daimler  Chrysler  Corporation 

•  Why  is  it  Extreme? 

-  Does  not  involve  bungee  cords;  no  relationship  to  Windows  XP  either...  © 

-  XP  adopts  well-known  software  development  practices  and  attempts  to  take 
them  to  their  logical  extremes 

•  Example:  The  “You  Aren’t  Gonna  Need  It”  (YAGNI)  Concept 

-  YAGNI  is  a  general  refrain  when  someone  suggests  building 
functionality  for  the  system  that  is  not  present  in  the  current 
requirements  set.  The  assumption  is  that  it  can  be  added  later  if  it 
becomes  necessary 

-  YAGNI  is  supposed  to  be  the  opposite  of  “Big  Design  Upfront  (BDUF) 

-  However,  remember  the  importance  of  diligent,  strategic  architecting 
and  design  we  described  earlier  to  prevent  random  software  failures 


BDUF  might  have  its  problems,  but  from  a  mission  assurance  perspective 
we  need  at  least  a  balanced  approach;  “extreme”  is  not  really  desirable 
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*  Source:  [Beck  2000] 


XP  Practices 

•  The  original*  XP  practices 

-  The  planning  game 

-  Small  releases 

-  Metaphor 

-  Simple  design 

-  Continuous  integration 

-  Continuous  testing 

-  Refactoring 

-  Pair  programming 

-  Collective  code  ownership 

-  40-hour  work  week 

-  On-site  customer 

-  Coding  standards 
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The  Planning  Game 

•  The  planning  game  is  a  metaphorical  name  for  requirements  engineering 
and  increment/iteration  planning 

-  It  is  essentially  a  meeting  where  the  team  is  working  through  a  stack  of  index 
cards  that  contain  the  user  stories 

-  Each  required  feature  is  described  and  elaborated  in  a  user  story  (another 
metaphor. . .) 

•  Responsibilities  during  the  planning  game* 

Customer  Developer 

Define  scope  of  the  release  Estimate  how  long  each  user 


Set  dates  and  times  of  release 


Define  order  of  delivery 


story  will  take 

Communicate  technical  impacts 
of  implementing  requirements 

Break  down  user  stories  into 
tasks  and  allocate  work 


★ 


Source:  [Baird  2002] 
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The  Planning  Game  -  2 


However,  the  needed  overall  systems  engineering  process  that  provides  the 
context  for  software  development  is  more  complex  [INCOSE  2003] 


It  is 

recursive 

and 

Iterative 


Process 

Input 


Mr.  Weasel  also  says. 


Process 

Output 


WeH^^Use^rcTyoiMead^Mak^lirec^responsibiHt^fo^h^Drogress^l 
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Small  Releases 


•  Start  with  the  smallest  feature  set,  release  early  and  often 

•  Duration 

-  Releases  may  be  provided  every  1-3  months 

•  Concerns 

-  The  earlier  mentioned  customer  problem 

•  The  need  for  excessive  participation  and  associated  responsibility  in  the 
planning  and  validation  of  these  releases  are  not  feasible 

-  Scaling  issue 

•  In  large  systems  it  might  be  difficult  to  come  up  with  a  finite,  incremental 
feature  set  to  field  tangible  releases  that  the  customer  could  appreciate 


Having  small,  internal  releases  is  a  good  engineering  practice  but  the 
customer  should  not  be  responsible  for  validating  these  releases 
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Metaphor 


•  Each  project  is  supposed  to  have  an  organizing  metaphor 

-  Metaphors  facilitate  the  dialog  between  the  user  and  developer 

-  Metaphors  serve  as  a  bridge  between  the  terminology  of  the  customer's 
domain  and  the  software  engineering  jargon 

-  A  metaphor  of  the  metaphor:  “Tribal  Language” 

•  Example  metaphor 

-  “Describing  an  agent-based  information  retrieval  system,  we  might  say  that 
this  program  works  like  a  hive  of  bees,  going  out  for  pollen  and  bringing  it 
back  to  the  hive”* 

•  This  practice  is  quite  benign  (as  opposed  to  “extreme”)  and  its  cost  is 
negligible.  However,  its  value  has  not  been  proven. 


Use  of  metaphors  do  not  seem  to  represent  any  risks 


Source:  [Stack  2008] 
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Simple  Design 


•  Keep  the  design  as  simple  as  possible  for  the  moment  and  don't  add 
features  that  are  not  needed  for  current  functionality 

-  The  reasoning  behind  this  practice  is  that  if  a  feature  is  not  valuable  now,  it 
is  not  worth  the  investment  until  it  becomes  valuable 

-  Simple  design  is  the  practice-level  implementation  of  the  earlier  introduced 
YAGNI  concept  and  the  avoidance  of  the  supposedly  bad  approach  of  BDUF 

•  Keeping  designs  simple  is  a  good  idea  in  general 

•  However,  the  operative  phrase  in  this  definition  is  “for  the  moment” 

-  Remember  Heisenbugs?  Prudent  consideration  for  all  the  overarching, 
nonfunctional  requirements  (like  reliability,  availability,  etc.)  requires 
extensive  upfront  design  and  thorough  follow-up  during  development 


A  shortsighted,  “extreme”  implementation  of  this  practice  might  lead 

to  a  mission  assurance  exposure 


49 


0 AEROSPACE 


Continuous  Integration  and  Continuous  Testing 


•  Continuous  Integration 

-  Integrate  with  the  whole  system  as  often  as  feasible 

•  Continuous  testing 

-  Unit  testing  and  acceptance  testing  are  alternating  according  to  the  rhythm  of 
the  process,  which  is  driven  by  the  duration  of  the  applied  timeboxes 

•  Unit  tests,  written  by  developers  to  test  functionality  as  they  implement  it 

-  Conceptually,  it  is  not  different  from  any  other  approaches 

-  In  agile  development  a  test-driven  strategy  is  preferred  where  the  unit 
test  suite  is  developed  before  coding  starts  and  the  execution  of  these 
tests  is  automated  -  no  particular  mission  assurance  exposure  here 

•  Acceptance  tests 

-  Tests  themselves  are  supposed  to  be  specified  by  the  user/customer 

-  User/customer  has  to  observe  all  tests  or  review  test  runs 

•  In  either  case  the  user/customer  is  expected  to  approve  test  results 
according  to  the  dictated  process’  rhythm 

•  However,  see  our  earlier  interim  progress  tracking  concerns: 

This  is  an  undue  burden  on  the  customer  -  continuous 
acceptance  tests  are  not  feasible  in  a  large  project 

0 AEROSPACE 
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Refactoring 

•  Refactoring  is  a  technique  to  improve  code  without  changing  functionality 

-  It  is  a  declared  XP  virtue  to  refactor  late  in  the  design  to  increase  performance 

•  Examples 

-  Repartitioning  the  code  to  smaller,  easier  to  maintain  chunks 

-  Renaming  some  variables  to  be  more  descriptive 

-  Re-evaluating  the  need  for  temporary  variables 

-  Extracting  common  behavior  into  a  single  code  segment 

-  Candidates  for  refactoring  may  be  found  via  the  “smell  test” 

•  Large  program  segments  or  classes 

•  Deeply  nested  code 

•  Long  parameter  list 

•  Presence  of  switch  (case)  statements 

•  Redundant  code  (e.g.,  a  class  that  does  not  seem  to  do  anything,)  etc. 

•  Risks 

-  Every  technique  that  changes  a  running  or  working  system  is  not  immune 

to  introducing  errors,  even  if  it  is  claimed  that  no  functionality  is  impacted  !i 

-  “Refactoring  in  the  small ”  can  be  helpful  but  “refactoring  in  the  large”  does  not 

make  sense  and  it  is  a  dangerous  practice _ 

^efactorln^misUTO^^iseTa^aTeplacemenno^rope^rchitectin^l 
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Pair  Programming 

Collaborative  programming  is  not  a  new  idea;  it  has  been  explored  before* 
Pair  programming  is  a  collaborative  technique  to  ensure  quality  code 

-  People  are  paired-up  at  a  workstation  and  working  together 

-  However ,  it  is  not  like  a  piano  duet  on  the  computer  keyboard  ® 


-  The  members  of  the  pair  have  different  roles  and  those  roles  may  change 

-  People  may  change  pairs  too  as  needed 

Pair  programming  is  one  of  the  most  debated  agile  practices 

-  Its  effectiveness  is  evaluated  on  the  following  three  dimensions  when  it  is 
compared  to  solo  programming:  Effects  on  quality,  duration,  and  effort. 

0 AEROSPACE 


*  See  [Nosek  1998] 


The  Effectiveness  of  Pair  Programming 

•  The  reported  results  are  based  on  a  meta-analysis  of  18  detailed 
studies* 

-  The  goal  of  a  meta-analysis  is  to  estimate  the  overall,  combined  effect 

-  Rigorous  meta-analysis  ensures  the  standardization  of  the  reported 
data  sets  and  provides  comparable  effect  estimates 

-  In  meta-analysis,  rather  than  computing  a  simple  mean,  more  weight  is 
assigned  to  studies  that  carry  more  information 

•  Authors  used  two  different  statistical  models;  we  present  their 
conclusions  for  a  so-called  “fixed-effects  model” 

-  A  fixed-effects  model  assumes  an  unknown  but  fixed  population 

-  All  18  studies  are  seen  as  data  drawn  from  the  same  population  and 
variances  between  individual  studies  are  viewed  as  results  of  subject 
variability 


Caveat:  Effort,  duration,  and  quality  are  not  well  defined  in 
general  and  are  operationalized  in  very  diverse  ways 
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* 


Source:  [Hannay  2009] 


Reported  Meta-analytic  Effects  of  Pair  Programming 


•  A  little  statistics 

-  The  standardized  measure  of  effect  size  is  Hedges’  g* 

•  An  effect  size  of  .5  indicates  that  the  mean  of  the  pair  programming 
group’s  distribution  is  half  a  standard  deviation  larger  than  the  mean  of 
the  reference  group’s  (the  solo  programmer’s)  distribution 
-  Effect  sizes  larger  than  1.0  are  “large,"  0.38  -  1.00  are  “medium," 
and  0-0.37  are  “small" 

•  Effect  sizes  from  the  meta-analysis 


Effect  on  Effect  Size  [g] 
Quality  +0.23 

Duration  +0.40 

Effort  -  0.73 


Description  of  Effect 

Small  significant  positive 
Low-medium  significant  positive 
Medium  significant  negative 


•  In  plain  English:  Minor  quality  improvement  and  some  schedule 
compression  can  be  achieved  at  the  price  of  somewhat  higher  cost 

In  even  plainer  English,  “Faster,  Better,  Cheaper”  does  not  work  here  either** 
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Mission  Assurance  Risk  in  Pair  Programming 


•  XP,  while  it  does  not  explicitly  forbid  formal  inspections,  treats  them  as 
redundant  and  unnecessary 

-  XP  proponents  claim  that  inspection  happens  all  the  time  through  pair 
programming 

-  However ,  that  pair  programming  is  a  general  improvement  over  formal 
inspections  (also  called  Peer  Reviews)  remains  unproven* 


•  Unique  benefits  of  formal  inspections 

-  Inspectors’  independence  from  the  creator  of  the  inspected  work  product 

•  “The  issue  is  closeness,  not  ability.  That’s  why  every  writer  needs  an  editor**” 
-  Note  that  we  used  the  term  “work  product,”  which  is  broader  than  “code” 

-  Knowledge  transfer  (although  should  not  be  treated  as  a  training  vehicle...) 

-  Improving  the  process,  like  adding  items  to  checklists,  recommending  tools  like 
a  static  code  analyzer,  recommending  changes  to  coding  standards,  etc. 

-  Reevaluating  assumptions  that  were  made  earlier  about  requirements 

-  Capturing  and  evaluating  quality  metrics,  identifying  common  problem  areas 

Despite  its  positive  impact  on  quality,  pair  programming  is  I 
not  an  acceptable  replacement  for  formal  inspections 
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Collective  Code  Ownership 


•  No  single  person  “owns”  a  module 

-  Any  developer  is  expected  to  be  able  to  work  on  any  part  of  the  code-base 
at  any  time 

•  In  theory  this  is  a  good  practice  regardless  of  the  used  software 
development  methodology 

•  Caveat:  In  reality  the  practice  does  not  scale  up 

-  There  are  limits  to  how  much  code  evolution  somebody  can  follow  real-time 

-  Also,  programmers  are  no  longer  equal  -  like  in  medicine,  high-level 
specialization  is  the  current  reality 

•  Specialization  examples 

-  Database  developers,  graphical  user  interface  (GUI)  developers, 
algorithm  developers,  networking  specialists,  infrastructure 
specialists  (formerly  called  “system  programmers’ ’),  etc. 

"^oMectiv^od^wnershipJf^ppIied^roperly^TarTpositiv^mpac^j 
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40-Hour  Work  Week 


•  Programmer  welfare  is  considered  important 

-  XP  development  is  considered  a  stressful  environment 

-  Programmers  should  go  home  on  time 

♦  Up  to  one  week  of  overtime  is  allowed  (Note  that  this  is  an  XP  guidance 
and  not  a  Human  Resources  (HR)  policy) 

♦  Consecutive  weeks  of  overtime  is  a  sign  that  the  process  might  be  failing 

•  My  take 

-  Not  just  XP  but  software  development  in  general  is  a  stressful  endeavor 

-  Everybody  should  go  home  on  time,  not  just  programmers  ...  © 


57 


0 AEROSPACE 


On-site  Customer 


•  According  to  this  practice,  the  developers  have  continuous  access  to 
a  real,  live  customer 

-  Note  that  this  is  different  (and  much  more  involved)  than  the  traditional 
Rapid  Application  Development  (RAD)  approach ,  where  the  customer 
primarily  participated  in  early  prototyping 

-  It  is  also  different  from  the  prevailing,  periodical  program  management 
reviews  where  customer  representatives  are  present 

•  In  case  of  large,  geographically  distributed  teams  this  expectation  is 
not  feasible 

-  Development  of  large  systems  usually  involves  geographically  distributed 
teams;  the  distributed  structure  of  the  organization  is  essentially  a  liability 
and  source  of  numerous  risks  that  need  to  be  dealt  with 

•  The  excessive  burden  on  government  personnel  makes  the  practice 
also  infeasible 

However,  the  main  risk  is  the  underlying  issue  that  the  customers 
are  now  made  implicitly  responsible  for  all  decisions  and  progress 
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Coding  Standards 


•  The  written  code  must  be  homogeneous 

-  One  should  not  be  able  to  tell  by  looking  at  the  code  who  on  the  team  wrote 
or  corrected  a  piece  of  it 

-  This  practice  is  closely  related  to  Collective  Code  Ownership 


Following  coding  standards  is  an  unconditionally  good  practice 
regardless  of  the  software  development  methodology  used 
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XP  Practice  Evolution  -  New  XP  Practices* 


•  The  planning  game 

-  Quarterly  Cycle  and  Weekly  Cycle  are  replacing  the  old  practice 

•  Small  releases 

-  Incremental  Deployment  and  Daily  Deployment  are  introduced 

•  Metaphor 

-  It  was  always  the  least  understood  practice  and  now  it  is  eliminated 

•  Simple  design 

-  Incremental  Design  and  Single  Code  Base  are  introduced 

•  Continuous  integration 

-  No  change 

•  Continuous  testing 

-  Emphasis  on  Test-First  Programming 

•  Refactoring 

-  Eliminated  as  a  formal  practice;  became  part  of  Incremental  Design 

•  Pair  programming 

-  No  change 
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*  See  [Beck  2004]  for  the  description  of  new  XP  practices 
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New  XP  Practices-2 


•  Collective  code  ownership 

-  It  is  now  called  Shared  Code 

•  40-hour  work  week 

-  Eliminated;  Energized  Work  and  Slack  replace  this  practice 

•  Energized  Work  is  a  reinterpretation  of  the  sustainable  pace  concept 

•  Slack  means  to  mark  things  that  can  be  dropped  if  you  get  behind 

•  On-site  customer 

-  Sit  Together ;  Whole  Team,  and  Real  Customer  Involvement  practices 
were  introduced 

•  Coding  standards 

-  Not  called  out  anymore  but  still  a  foundation  of  Shared  Code 

•  There  are  more  new  practices,  a  new  value,  and  several  new  principles 
but  we  were  only  focusing  on  the  evolution  of  the  original  12  practices 
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Mission  Assurance  Consequences 


•  A  reviewer’s  opinion  about  the  2nd  edition  of  Beck’s  book: 

-  "...  the  2nd  edition  describes  a  new  process  that  is  different  from  the  process 
Beck  describes  in  the  first  book,  it  seems,  he  has  invented  a  new  process 
(based  on  his  experience  with  XP)  and  gave  it  the  same  name”* 

•  The  importance  of  process  documentation  and  use  of  standards 

-  XP  is  a  good  example  of  how  fluid  the  agile  field  still  is  and  how  difficult  it  is 
to  pin  down  specific  practices 

-  High-quality,  detailed  process  documentation  is  needed  to  mitigate  upfront 
agile  process  ambiguities;  carrying  out  the  oversight  function  is  very  difficult 
without  documented,  agreed-upon  terminology  and  processes 

-  The  customer  must  understand  that  (s)he  will  only  get  what  (s)he  explicitly 
asks  for;  after  the  contract  is  signed,  the  customer  is  at  the  mercy  of  the 
contractors  and  will  be  separately  charged  for  every  request  that  is  deemed 
to  be  “new” 

-  For  a  more  detailed  analysis  see  the  earlier  mentioned  report  [Eslinger  2006] 


Use  of  standards  is  one  of  the  most  effective  tools  for  the 
customer  to  go  on  record  with  process-related  expectations 


62 


Source:  [Stansell  2004] 


Top  12  Reasons  Named  in  2010  for  Adopting  Agile* 


•  Accelerate  time  to  market 

•  Enhance  ability  to  manage  changing  priorities 

•  Increase  productivity 

•  Enhance  software  quality 

•  Improve  alignment  between  information  technology  (IT)  and  business 
objectives 

•  Improve  project  visibility 

•  Reduce  risk 

•  Simplify  development  process 

•  Enhance  software  maintainability  and  extensibility 

•  Improved  team  morale 

•  Reduce  cost 

•  Improve  and  increase  engineering  discipline 

Some  of  these  expectations  are  clearly  counter-intuitive,  showing 
a  lack  of  true  understanding  of  these  methodologies 
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Top  12  Concerns  in  2010  About  Adopting  Agile* 


•  Loss  of  management  control 

•  Lack  of  upfront  planning 

•  Management  opposed  to  change 

•  Lack  of  documentation 

•  Lack  of  predictability 

•  Lack  of  engineering  discipline 

•  Development  team  opposed  to  change 

•  [Lack  of]  engineering  talent 

•  Inability  to  scale 

•  Regulatory  compliance 

•  Reduced  software  quality 

•  Other 


It  is  not  on  the  list,  but  one  of  the  main  concerns  should  be  lack  of 
consistent  metrics  and  reliable  data  to  verify  if  any  of  the  objectives 
stated  on  the  previous  slide  have  been  met 
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Agile  Software  Development  from  a  Commercial 
Perspective 

•  Using  agile  software  development  is  a  business  strategy,  based  on  a 
particular  value  proposition 

•  Choosing  a  development  method  should  be  based  on  the  home  grounds 
of  the  organization  and  project,  characterized  by  the  following  factors* 

-  Project  size  (expressed  by  the  number  of  development  personnel  involved) 

-  Criticality  (loss  due  to  impact  of  defects) 

-  Level  of  software  understanding  in  development  personnel 

-  Dynamism  (%  requirements-change/month) 

-  Culture  (thriving  on  chaos  vs.  preferring  order) 

•  When  these  metrics  are  considered,  agile  software  development 
certainly  seems  to  be  a  promising  approach  for  small,  low-criticality 
projects  with  rapidly  changing  requirements,  where  the 
organization’s  culture  embraces  high  degrees  of  freedom,  and  the 
developers  are  highly  experienced 

Unfortunately,  the  applicability  of  these  methods  outside  of 
the  above  described  home  grounds  has  not  yet  been  proven 
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*  [Boehm  2004] 


AEROSPACE 

Assuring  Space  Mission  Success 


Is  Agility  Really  the  Answer  to  Fix  the  Broken 
Acquisition  System? 


How  do  We  Know  that  it  is  Broken? 


JCIDS 


4 


•  Defense  Acquisition  Performance  Assessment  (DAPA)  summary  in  2006 

-  “As  early  as  1971  it  has  been  identified  that  [defense]  acquisition  processes 
have  significant  shortcomings  leading  to  loss  of  confidence  by  congress  and 
the  defense  community ” 

-  “Many  improvements  to  the  DOD’s  acquisition  system  have  been  made  as  a 
result  of  past  reviews  . . .  However ,  the  ability  to  deliver  operational 
performance  of  major  systems  within  predicted  cost  and  schedule  has 
not  improved  over  the  last  20  years” 
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Selected*  DAPA  Recommendations  in  2006 


•  Replace  the  Joint  Capability  Integration  Development  System  (JCIDS) 
with  a  new,  two-year  recurring  planning  process  based  on  the  15-year 
extended  plans  submitted  by  combatant  commands 

•  Stabilize  the  Planning,  Programming,  Budgeting,  and  Execution 
(PPBE)  process 

•  Introduce  a  new  requirements  process  with  2-year  duration 

•  Establish  a  distinct,  stable  Program  Funding  Account 

•  Increase  program  predictability 

•  Program  all  accounts  to  a  high,  80/20  confidence  level 

•  Establish  very  early  a  realistic  capability  delivery  rate 

•  Establish  very  early  all  test  plans 

-  Complete  Test  &  Evaluation  Management  Plan  (TEMP)  and  Initial 
Operational  Testing  &  Evaluation  Plan  (IOT&EP)  priorto  Milestone  B 

Clearly,  the  DAPA  panel  valued  stability  and 
predictability  as  opposed  to  agility 

*  There  were  more  recommendations  but  those  did  not  have  potential  agile 
69  implications 


Acquisition  Problems  Identified  in  2011  by  the 
Government  Accountability  Office  (GAO)* 

>  Alternatives  not  considered 

-  Clearly,  no  relationship  to  agile  development 

’  Funding  unstable 

-  Actually,  agile  development  is  supposed  to  be  an  adaptive  mechanism  that 
might  be  helpful  in  dealing  with  unstable  funding,  but  only  at  the  price  of 
delaying  or  dropping  requirements 

*  Inadequate  contracting  strategy 

-  The  report  is  referring  to  the  failure  of  Total  System  Performance 
Responsibility  (TSPR)  and  lack  of  evolutionary  strategies  in  certain 
acquisitions;  neither  has  agile  software  development  implications 

>  Inadequate  contractor  oversight 

-  This  concern  is  also  related  TSPR;  While  some  agile  principles  would  embed 
more  government  personnel  in  the  development  process,  due  to  lack  of 
contracting  rigor  this  involvement  would  be  costly  and  ineffective 

-  Also,  increasing  the  acquisition  work  force  has  been  suggested,  but  in 
the  current  climate  of  drastic  budget  cuts  it  is  not  feasible 

•  $148-$178B  DOD  cuts  planned  between  2012  and  2016** 


[Chaplain  2011],  **[Weisgerber  2011] 


More,  GAO-identified  Acquisition  Problems 

•  Optimistic  cost  and  schedule  estimates 

-  The  operative  word  seems  to  be  “ optimistic which  has  nothing  to  do  with 
the  details  of  development  methodologies.  Additionally,  due  to  the  difficulties 
with  macro-estimation  in  agile  development,  one  can  expect  further 
dissatisfaction  with  the  accuracy  of  cost  and  schedule  estimates 

•  Requirements  unstable 

-  Because  of  its  adaptive  nature,  agile  development  is  supposed  to  help  with 
handling  unstable  requirements.  However,  regardless  of  the  implemented 
agile  project  management  strategy,  volatile  requirements  will  yield  inaccurate 
cost  and  schedule  estimates,  ultimately  resulting  in  customer  dissatisfaction 

•  Software  needs  poorly  understood 

-  This  is  also  a  requirements  and  early  architecting  issue.  Again,  selected 
agile  development  practices  do  facilitate  the  gradual,  more  effective 
discovery  of  software-level  requirements,  but  still,  software  estimates, 
particularly  the  early  ones,  will  be  grossly  inaccurate 

•  Technology  immature 

-  This  has  not  been  a  software  issue  on  the  reviewed  acquisitions 

^gilit^n^gH^oftwar^evelopmen^tilM^io^eerT^^^h^nswer^l 
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What  May  the  Future  Bring? 


•  The  most  significant  recent  directive  by  Congress  that  could  shape  the 
future  of  defense  acquisitions  is 

-  Public  Law  111-84,  The  National  Defense  Authorization  Act  for  Fiscal 
Year  2010;  Section  804.  Implementation  of  New  Acquisition  Process 
for  Information  Technology  Systems 

The  central  question  is  this:  Is  it  true  that  the  law  directs  the 
incorporation  of  agile  methodologies  in  DOD  software  acquisitions? 

0 AEROSPACE 


72 


Does  Section  804  Direct  the  Incorporation  of  Agile 
Methodologies  in  DOD  Software  Acquisition? 

•  Why  are  we  even  asking  the  question? 

-  Quotes  from  the  November  14-15,  2011  National  Defense  Industry 
Association  (NDIA)  Agile  Scrum  Workshop’s  invitation 

•  “The  law  [Section  804]  directs  the  incorporation  of  Agile  methodologies  in 
DOD  software  acquisition  ...  Agile  cannot  fail.  Unequivocally,  Agile  cannot 
fail.” 

•  However,  what  Section  804  actually  requires  is  an  acquisition  process 
with  the  following  characteristics: 

-  Early  and  continual  involvement  of  the  user 

-  Multiple,  rapidly  executed  increments  or  releases  of  capability 

-  Early,  successive  prototyping  to  support  an  evolutionary  approach 

-  Modular  open  systems  approach  (MOSA) 

•  Again,  Section  804  requires  a  new  acquisition  process  but  congress 
cannot  (and  should  not)  legislate  a  software  development  process 

^^Th^wordin^^ndee^nspimdTyagiITideas^uUh^onnect!oM^^T| 
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Conclusions  - 1 


•  Continuing  problems  in  the  software  enterprise  (earlier  the  symptom 
was  called  the  “software  crisis”)  pushed  organizations  to  continuing 
experimentation  with  new  development  methods 

-  Part  of  this  experimentation  is  manifested  in  the  rediscovery  and 
sometimes  just  renaming  of  known  processes 

-  Experimentation  is  further  fueled  by  the  “bandwagon  effect” 

-  Unfortunately,  there  is  no  sufficient  data  with  acceptable  quality  available 
to  properly  characterize  the  emerging  agile  methods  and  establish  a 
reliable  performance  baseline 


In  the  meantime,  defense  acquisitions  of  software-intensive  systems 
are  still  struggling  and  there  is  no  effective  solution  in  sight 

-  The  demand  for  bigger  and  more  sophisticated  weapon  systems  is 
constantly  increasing  while  the  scaling  problem  of  processes  and  the 
management  of  the  continuously  growing  scope  are  not  resolved 

-  Also,  a  tendency  for  blind  copying  of  industry  practices  is  present  due  to  a 
persistent  opinion  that  “industry  knows  what  to  do  and  we  should  just 
adopt  industry  practices” 

•  Unfortunately,  the  associated  risks  are  not  well  understood  and  in  some 

cases  are  explicitly  covered  up  ^  _ _ _ _ 
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Conclusions  -  2 


•  What  APO  personnel  needs  to  do 

-  Continuously  educate  itself  on  the  emerging  development  methods 

-  In  the  contracting  phase  must  insist  on  the  use  of  robust  development 
standards 

•  The  government  should  not  settle  for  vague  references  to  agile 
programming;  it  must  insist  on  a  detailed  software  development  plan 
(SDP)  that  fully  characterizes  all  planned  life  cycles,  their  internal 
relationships,  and  the  planned  implementation  details  of  all  life-cycle 
processes  and  associated  activities 

•  Mission  success  criteria  and  synergy  with  mission  assurance  needs  must 
to  be  used  to  validate  the  SDP  before  acceptance  for  the  contract 

-  In  the  contract  monitoring  phase  must  implement  an  effective  mission 
assurance  program 

•  Mission  assurance  is  essentially  an  ingrained  instrumentation  of  the 
development  process;  it  is  a  necessity  and  must  not  be  allowed  to  be 
viewed  by  the  development  organization  as  a  “nice-to-have,”  negotiable 
feature 
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Conclusions  -  3,  or  What  You  Really  Need  to  Remember... 


“The  temptation  to  'cut  corners,'  even  in  the  name  of  being  efficient  or 
'expedient,'  is  ever-present,  especially  in  a  global  business  that  is 

economically  unforgiving... 

That  Is  why  'getting  it  right'  must  be  a  24/7  commitment" 

~  Dr.  Wanda  Austin,  President  and  CEO,  The  Aerospace  Corporation 
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Acronyms 


APO 

Acquisition  Program  Office 

ATIP 

Aerospace  Technical  Investment  Program 

BDUF 

Big  Design  Up  Front 

CEO 

Chief  Executive  Officer 

CMMI 

Capability  Maturity  Model  Integration 

COTS 

Commercial  Off-the-shelf 

DAPA 

Defense  Acquisition  Performance  Assessment 

DoD 

Department  of  Defense 

FAR 

Federal  Acquisition  Regulation 

FFRDC 

Federally  Funded  Research  &  Development  Center 

GAO 

General  Accountability  Office 

GUI 

Graphical  User  Interface 

IBM 

International  Business  Machines 

IID 

Iterative-Incremental  Development 

IOT&EP 

Initial  Operational  Testing  &  Evaluation  Plan 

IT 

Information  Technology 

IV&V 

Independent  Verification  &  Validation 

JCIDS 

Joint  Capabilities  Integration  &  Development  System 

JROC 

Joint  Requirements  Oversight  Council 
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KDSI 

Thousand  Delivered  Source  Instructions 

LTCD 

Long  Term  Capability  Development 

MOSA 

Modular  Open  System  Architecture 

NDIA 

National  Defense  Industry  Association 

OSD 

Office  of  the  Secretary  of  Defense 

OT&E 

Operational  Test  &  Evaluation 

PPBE 

Planning,  Programming,  Budgeting  &  Execution 

RAD 

Rapid  Application  Development 

RUP 

Rational  Unified  Process 

SE&I 

Systems  Engineering  &  Integration 

SETA 

Systems  Engineering  and  Technical  Assistance 

SEU 

Single  Event  Upset 

SW 

Software 

TOO 

Total  Cost  of  Ownership 

TEMP 

Test  &  Evaluation  Management  Plan 

TQM 

Total  Quality  Management 

TSPR 

Total  System  Performance  Responsibility 

XP 

extreme  Programming 

YAGNI 

You  Aren’t  Gonna  Need  It 
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Representative  Agile  Software  Development  Methods 


•  Agile  UP  (Agile  Unified  Process)  [Ambler  2006] 

•  ASD  (Adaptive  Software  Development)  [Highsmith  2000] 

•  Crystal  Clear  [Cockburn  2004] 

•  DSDM  (Dynamic  Systems  Development  Method)  [Stapleton  2003] 

•  Ess  UP  (The  Essential  Unified  Process)  [Jacobson  2006] 

•  XP  (extreme  Programming)  [Beck  2000],  [Beck  2004] 

•  FDD  (Feature-Driven  Development)  [Palmer  2002] 

•  Lean  Software  Development  [Poppendieck  2003,  Poppendieck  2006] 

•  MSF  (Microsoft  Solution  Framework)  for  Agile  Development  [MSF  2006] 

•  Scrum  [Schwaber  1995] 
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